Only re-dump package metadata when its dumpable content changed - #1803

Re-dumping v2 metadata used to be driven almost entirely by crawledAt: any crawl re-staled the package, so the monthly "crawl all" flooded the dumper with packages whose files had not changed. This makes re-dumps content-driven instead — a crawl marks the package only when it actually changed dumpable content: a version created or mutated (which also covers the default-branch toggle, the source/dist URL rewrite and the abandoned-state change, all of which surface as a VersionUpdatedResult), an auto-recovered version, or an auto-soft-deleted one. An unchanged re-crawl no longer marks anything.

Marking is non-destructive

Marking used to null dumpedAtV2, which loses requests: V2Dumper::dump() records dumpedAtV2 for a whole batch only at the end, so a mark landing between a package's hydration and that write was silently overwritten. crawledAt masked that for the Updater, but SecurityAdvisoryUpdateListener and FilterListEntryUpdateListener never touch crawledAt — so a newly published advisory could stay missing from p2/ until an unrelated crawl. That was a live bug, and invisible to any file-level metric since no file is written.

A new package.dumpRequestedAt column records the request instead, and staleness becomes dumpedAtV2 IS NULL OR dumpRequestedAt >= dumpedAtV2. $dumpTime is captured before hydration, so a mark landing mid-run leaves the package stale and it is picked up next run. >= rather than > because both columns hold whole seconds: a same-second mark is ambiguous and must resolve towards re-dumping.

Two levels of request:

• Package::markForDump() — "re-dump if the content changed". Every ordinary content-change path uses this; writeV2File() still compares content and early-returns if it matches.
• Package::forceDump() — additionally nulls dumpedAtV2, which writeV2File() reads as "write this regardless", re-uploading byte-identical files to bust the CDN cache and re-run the replica write. Reserved for the deliberate "make it publish again" paths: Updater::FORCE_DUMP (manual update button, unfreeze, repo URL rewrite) and Package::unfreeze(). It always marks as well, so an in-flight dump run can at worst cost the forcing, never the request.

VersionRepository::softDelete() / recover() now mark directly rather than relying on the scheduled crawl: pulling a version is the security path, and a package whose repository 404s never reaches the end of Updater::update(), which would leave a pulled version published indefinitely. remove() marks directly too, but not for rows that were already soft-deleted — those were excluded from the dumped metadata already, so hard-purging them changes no bytes.

Scheduler::scheduleUpdate() keys its dedup on (packageId, update_source_dist_url, delete_before), so a pending plain job used to swallow a force_dump request entirely. It now carries the intent both ways: onto an already-queued job, and onto the replacement job when a scheduled-for-later job is cancelled in favour of an immediate one.

Convergence fixes in the Updater

updateInformation() rebuilt a dev version on every crawl, forever, whenever its rebuild trigger could not agree with what the reconciliation stores — bumping Version::updatedAt (= published-time for dev rows), i.e. a real CDN upload + replica write + purge each time. Three causes, all fixed:

• the replacement was only written while transitioning to abandoned, so a standing mismatch never self-healed. It is reconciled on every crawl that sees upstream abandonment now (composer.json wins; a UI-only abandon is untouched);
• the trigger compared the raw upstream value against the sanitized stored one, so any replacement sanitize() alters could never match. Both sides go through normalizeReplacementPackage() now;
• the trigger fired for every dev version while the reconciliation is scoped to the default branch, so a non-default branch declaring abandonment the default branch disagrees with never converged. Both are default-branch-scoped now — version rows carry no abandoned state of their own, Version::toV2Array() reads it off the package at dump time.

Instrumentation and the transitional net

The crawledAt staleness clause is deliberately unchanged, so selection remains a strict superset and there is no regression. Dropping it is a follow-up, gated on a new gap-detector: metadata_dump.file{result:written, requested:false} counts content changes that only crawledAt caught, i.e. a change path that fails to mark, and must sit at ~0 first. Its inverse {result:skipped, requested:true} catches paths that mark more eagerly than needed. result:created splits out first writes of a missing local file so they do not pollute the gate, and the legacy filemtime >= 1606210609 re-dump branch — which re-verified any p2 file whose mtime predates 2020-11-24 — is removed for the same reason.

Schema

Two migrations, one on each side of the deploy, because the stale query pins its index by name with USE INDEX:

• 2026_08_package_dump_requested_at.sql — before the deploy: adds dumpRequestedAt and dumped2_requested_crawled_frozen_idx.
• 2026_08b_drop_dead_dump_columns.sql — after the deploy: drops dumped_idx, the dead v1 dumpedAt column (the v1 dumper is gone and its readers had no callers), and the superseded dumped2_crawled_frozen_idx.